Displaying image data behind surfaces

ABSTRACT

Examples are disclosed herein that relate to displaying image data configured to appear behind a real-world surface. One example provides, on a computing device including a display, a method including obtaining depth data representing a real-world scene, identifying a real-world surface of the real-world scene via the depth data, obtaining volumetric image data and surface image data, the volumetric image data configured to appear as being located in a volume behind the real-world surface, receiving a user input configured to remove an area of surface image data corresponding spatially to the real-world surface, and displaying at least a portion of the volumetric image data in a region in which the area of surface image data was removed.

BACKGROUND

Display technologies may allow a user to experience immersive virtualenvironments and/or a mix of real and virtual environments. For example,some computing devices may include see-through displays that allow thepresentation of augmented reality imagery via the display of virtualobjects superimposed over a real-world environment.

SUMMARY

Examples are disclosed herein that relate to displaying image dataconfigured to appear behind a real-world surface. One example provides,on a computing device including a display, a method including obtainingdepth data representing a real-world scene, identifying a real-worldsurface of the real-world scene via the depth data, and obtainingvolumetric image data and surface image data, wherein the volumetricimage data is configured to appear as being located in a volume behindthe real-world surface. The method further includes receiving a userinput configured to remove an area of surface image data correspondingspatially to the real-world surface, and displaying at least a portionof the volumetric image data in a region in which the area of surfaceimage data was removed.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example use environment for an augmented reality displaydevice, and illustrates example augmented reality imagery.

FIG. 2 shows example volumetric image data revealed by a user input.

FIG. 3 shows example augmented reality image data displayed whileapplying real-world lighting effects.

FIG. 4 shows example augmented reality image data having lightingeffects based on virtual and real-world light sources.

FIG. 5 shows example augmented reality image data having lightingeffects based on a user interaction removing a region of volumetricimage data.

FIG. 6 shows another example of volumetric image data revealed by a userinput.

FIGS. 7A and 7B show an example method of displaying image data on anaugmented reality display system.

FIG. 8 shows a block diagram of an example augmented reality displaysystem.

DETAILED DESCRIPTION

Current augmented reality imagery may take the form of surfaceaugmentations displayed over real-world surfaces. Such augmentations maychange the appearance of the real-world surface, but may not beinteractive to reveal additional imagery configured to appear to belocated within or behind the real-world object. Accordingly, examplesare disclosed herein that relate to displaying image data configured toappear as being located behind real-world surfaces. As described in moredetail below, the disclosed examples may allow volumetric image data toinitially appear to be concealed behind a surface and then revealed byuser inputs. This may provide a visually stimulating interactiveexperience of virtually “digging into” a real world object that is notprovided by surface augmentation alone. Further, lighting effects basedupon both real and virtual features may be applied to such image data.

FIG. 1 shows an example use scenario in which a user 100 is wearing anaugmented reality display device 102 that presents virtual image datatogether with a see-through view of a real-world scene 104 within anaugmented reality field of view 106. In this example, the augmentedreality display device 102 is displaying a virtual object 108 positionedunder a table within the real-world scene 104. As described in moredetail below, the augmented reality display device 102 is alsodisplaying an outermost portion of volumetric image data, which is alsoreferred to herein as surface data. This surface data is locatedspatially at the surface of one or more real-world objects in the scene,but may in some instances not be rendered such that it does not occludea view of the real world surface. Further, the surface image data can beremoved by user input to reveal rendered volumetric image dataapparently located within the volumes behind the surfaces.

The augmented reality display device 102 may include one or moreoutward-facing image sensors configured to acquire image data of thereal-world scene 104. Examples of such image sensors include, but arenot limited to, depth sensor systems (e.g. time-of-flight or structuredlight camera(s)), visible light image sensors, and infrared imagesensors. The augmented reality display device may obtain a virtualrepresentation of the real-world scene 104 for the presentation ofaugmented reality images. For example, the augmented reality displaydevice may obtain a three-dimensional mesh of the real-world scene 104constructed from depth data acquired via an on-board depth sensor (e.g.by using a simultaneous localization and mapping method). In otherexamples, the augmented reality display device may obtain previouslyacquired and stored depth data of the real-world scene 104, eitherstored locally or remotely.

After obtaining the virtual representation (e.g. three-dimensional mesh)of the real-world scene 104, the augmented reality display device 102may identify one or more real-world surface(s) of the real-world scenevia the virtual representation. For example, the augmented realitydisplay device 102 may identify couch surfaces 110, table surfaces 112,and a wall surface 114, as non-limiting examples of identifiablereal-world surfaces. Such surfaces may be identified in any suitablemanner, such as by classification of real-world objects using imageand/or depth data, or by user-designated labels.

The augmented reality display device 102 may obtain image datacorresponding to the virtual representation of the real-world scene 104.The image data obtained may include volumetric image data correspondingspatially to volumes behind the surfaces, and also surface image data,which is volumetric image data located at the physical surface. In FIG.1, the augmented reality display device 102 may obtain surface imagedata corresponding to the couch surfaces 110, table surfaces 112, wallsurface 114, and any other suitable identified real-world surfaces, andalso volumetric image data located behind any of these surfaces and/orothers. The volumetric image data may represent the real-world scene 104in any suitable manner, such as in the form of repeating geometric (e.g.cubic or other suitable shape) units configured to fill the volumesbehind the surfaces.

The surface image data is displayed by the augmented reality displaydevice 102 as aligned over the real-world surfaces in the real-worldscene, such that the user 100 views the surface image data as thecurrently perceived real-world surfaces. As such, the augmented realitydisplay device 102 may be configured to track movements of the user 100(e.g. head movements, body movements, and/or eye movements) tocontinually update display of the surface image data to help ensureproper alignment between the displayed surface image data and theviewable real-world surfaces. Such tracking and updating may beperformed in any suitable manner, including but not limited to viamotion sensors, gaze detectors, and/or outward-facing cameras disposedon the augmented reality display device 102.

The volumetric image data is configured to appear as being located in avolume behind a real-world surface. As examples, volumetric image datamay be located to appear as occupying the interior volume of the couchbehind couch surfaces 110, within the interior volume of the tablebehind table surfaces 112, in the space beneath the table (e.g.displayed virtual object 108), and/or behind the wall surface 114. Thus,some portions of obtained volumetric image data may not initially bevisible, such as portions concealed by the real-world surfaces and thesurface image data, while other portions may be initially visible, suchas virtual object 108.

The augmented reality display device 102 may recognize various userinteractions or other events to reveal volumetric image data initiallyconcealed behind a surface. FIG. 2 illustrates an example of volumetricimage data 200 which was not visible in FIG. 1 being revealed by a usergesture, represented via a hand 204. The hand 204 is shown as revealingthe volumetric image data 200 behind couch surface 110. In this example,the gesture input is shown as controlling a virtual cursor 206. Thevirtual cursor 206 thus acts as a “digging” tool allowing the user 100to virtually dig through the couch surface 110. Via the virtual cursor206, the gesture input removes an area of surface image data thatcorresponds spatially to the couch surface 110. Upon “digging” thesurface image data away, the volumetric image data 200 is displayed inthe region in which the area of the surface image data was removed, thusgiving the appearance that an interior volume behind the couch surface110 has been revealed. It will be understood that any suitable event maytrigger the revealing of the volumetric image data 200, such asdeployment of virtual explosives, etc.

In some implementations, the volumetric image data may be rendered priorto display. In such implementations, the volumetric image data may berendered when initially produced or otherwise obtained (e.g. when cubesor other shapes corresponding to various virtual structures, such asdirt, rocks, water, air, and the like are first associated withlocations in the real-world scene). Surface image data may be leftunrendered where it is desired to initially conceal the volumetric imagedata, or may be rendered where it is desired for the surface image datato be initially viewable. In other implementations, the volumetric imagedata 200 may remain initially unrendered, and may be rendered in realtime when revealed by user input or other event.

As mentioned above, the augmented reality display device 102 may beconfigured to apply lighting effects. Such effects may be used to makereal-world surfaces appear as being lit by virtual light sources, andalso to make virtual objects appear as being lit by real-world lightsources. To apply real-world lighting effects, as one example, theaugmented reality display device 102 may model virtual light sourcesthat match the locations and lighting characteristics of real-worldlight sources. As another example, one or more predetermined virtuallight source(s) may be utilized, e.g. a top-down light source.

FIGS. 3-5 illustrate examples of lighting effects that may be applied toaugmented reality image data. First, FIG. 3 shows a view of the virtualobject 108 of FIG. 1, being displayed as volumetric image data beneaththe table surface 112. As shown, the table surface 112 casts a shadow300 due to overhead lighting in the room. As such, the virtual object108 may be displayed based on such real-world lighting effects, e.g. byapplying the shadow 300 cast by the table surface 112 onto the virtualobject 108. The shadow 300 may be displayed by virtually casting theshadow onto surface image data (whether the surface image data itself isrendered or unrendered) corresponding to the floor surface. The surfaceimage data also may be augmented using local virtual light sources. FIG.4 shows an example virtual torch 302, for example, as placed by a user,that may alter the appearance of the virtual object 108 and the shadow300. Light from a virtual light source may be applied to a real-worldsurface as well, e.g. by applying the lighting effect to surface imagedata.

FIG. 5 shows an example display of image data after receiving a userinput removing an area 500 of surface image data corresponding spatiallyto the table surface 112. In this case, volumetric image data 502corresponding to the interior volume behind the table surface 112 isdisplayed upon removal of the area 500, e.g. via user interaction.Additionally, the augmented reality display device 102 may applyreal-world lighting effects illuminating the shadow 300 and the virtualobject 108, as if real overhead light were passing through area 500.This may provide a realistic effect that a portion of the table has beenremoved.

FIG. 6 depicts another example of displaying volumetric image databehind a real-world surface. Here, the user 100 views a real-world shelf600 mounted on the wall surface 114 of the real-world scene 104. Theuser 100 performs a gesture interaction via a hand 604 to remove an areaof surface image data corresponding spatially to the wall surface 114,and the augmented reality display device 102 displays volumetric imagedata 606 in a region within the removed area. As mentioned above, thedisplayed data may be pre-rendered or rendered in real time. Further,the user 100 may continue to interact with the volumetric image data 606to reveal additional portions of the volumetric image data 606 andprovide an effect of “digging” through the wall surface 114.

The augmented reality display device 102 may be configured to displaythe volumetric image data 606 based on the user's position to provide aparallax effect. For example, as the user 100 moves around the shelf600, the augmented reality display device 102 may track the position ofthe user 100 and change the apparent location and perspective of thevolumetric image data 606 based on the user's perspective, thusproviding a realistic sense of depth.

FIGS. 7A-7B show a flow diagram illustrating an example method 700 fordisplaying image data via a augmented reality display device. Method 700includes, at 702, obtaining a three-dimensional mesh representation of areal-world scene. The three-dimensional mesh representation may begenerated via image data acquired of the real-world scene, as shown at704. For example, the mesh may be generated via a simultaneouslocalization and mapping algorithm as a user of an augmented realitydisplay device comprising suitable sensors (e.g. one or more depthcameras, two-dimensional image sensors, motion sensors, globalpositioning system sensors, etc.) moves within the real-world scene.Alternatively or additionally, a previously-acquired three-dimensionalmesh representation may be retrieved from storage, either locally orremotely, as shown at 705. Method 700 further includes, at 706,identifying a real-world surface of the real-world scene, and at 708,obtaining volumetric image data including surface image data. Asdescribed above, the volumetric image data is configured to appear asbeing located in a volume behind the real-world surface, and the surfaceimage data, which is an outermost portion of the volumetric image data,corresponds spatially to the real-world surface. The surface image datamay be unrendered to hide the underlying volumetric data until revealedvia receipt of a user input removing the surface image data.

The volumetric image data and the surface image data may be constructedbased on the three-dimensional mesh representation of the real-worldscene, as shown at 710. For example, as described above, the volumetricimage data may be constructed to fill determined volumes in thereal-world scene. The volumetric image data may be constructed asrepeating units configured to fill the volumes, such as blocks that eachhave a predetermined appearance, or may be constructed in any othersuitable manner. For example, the volumetric image data also may takethe form of voxels. In other examples, the volumetric image data may beobtained from a remote device, as shown at 712, rather than constructedremotely. In such examples, the mesh representation of the real-worldscene may be sent to the remote device. The volumetric image dataoptionally may be pre-rendered, at 714, and the surface image data mayoptionally be left unrendered, at 716.

Continuing with FIG. 7B, method 700 further includes applying lightingeffects to the image data, at 718. For example, lighting effects may beapplied that are based on real-world lighting conditions, at 720, aswell as based on virtual light sources that are a part of the augmentedreality image data, at 722. Lighting effects that are based onreal-world light sources may be applied, for example, by modeling thereal-world light sources with virtual light sources positioned atsimilar locations and having similar light characteristics. Method 700additionally may include, at 724, applying virtual lighting effects tounrendered surface image data. This may give the appearance that virtuallighting sources effect the light cast on real-world objects. Method 700further may include, at 722, applying a shadow cast by the real-worldsurface onto the volumetric image data, as illustrated by example inFIG. 3, and at 724, applying virtual lighting effects within the shadow,as illustrated by example in FIG. 4.

A user may choose to interact with the images. As such, method 700further includes, at 726, receiving a user input configured to remove anarea of augmented reality image data. The area removed may be renderedvolumetric or surface image data (where the surface image data is theoutermost extent of the volumetric image data that corresponds spatiallyto the real-world surface), or the area may be unrendered surface imagedata. Any suitable user input may indicate the removal of surface orvolumetric image data, including but not limited to a gesture input, aneye gaze input, a voice input, and/or an input via a virtual cursor.

Method 700 includes, at 730, displaying at least a portion of thevolumetric image data in a region in which the area of the surface imagedata was removed, such as the volumetric image data 200 displayed asappearing behind the couch surface 110 or the volumetric image data 606behind the wall surface 114. Where the area removed includes unrenderedsurface image data, removing the area to expose volumetric data mayproduce the effect of digging into a real-world surface to reviewunderlying structures. Further, lighting effects may be applied to thenewly-displayed volumetric image data based upon any relevant real-worldand/or virtual light sources, as indicated at 732.

FIG. 8 shows a block diagram of an example augmented reality displaysystem 800. Display system 800 includes one or more lenses 802 that forma part of a see-through display subsystem 804, such that images may bedisplayed via lenses 802 (e.g. via projection onto lenses 802, waveguidesystem(s) incorporated into lenses 802, and/or in any other suitablemanner). Display system 800 further includes one or more outward-facingimage sensors 806 configured to acquire images of a real-worldenvironment being viewed by a user, and may include one or moremicrophones 808 configured to detect sounds, such as voice commands froma user or ambient sounds. Outward-facing image sensors 806 may includeone or more depth sensor(s) and/or one or more two-dimensional imagesensor(s) (e.g. RGB image sensors and/or infrared image sensors). Inother examples, augmented reality display system 800, may displayaugmented reality images via a viewfinder mode for an outward-facingimage sensor, rather than via a see-through display subsystem.

Display system 800 may further include a gaze detection subsystem 810configured to detect a gaze of a user for detecting user inputinteracting with displayed image data, for example when display system300 is implemented as a head-mounted display system, as mentioned above.Gaze detection subsystem 810 may be configured to determine gazedirections of each of a user's eyes in any suitable manner. For example,in the depicted embodiment, gaze detection subsystem 810 comprises oneor more glint sources 812, such as infrared light sources configured tocause a glint of light to reflect from each eyeball of a user, and oneor more image sensor(s) 814, such as inward-facing sensors, configuredto capture an image of each eyeball of the user. Changes in the glintsfrom the user's eyeballs and/or a location of a user's pupil asdetermined from image data gathered via the image sensor(s) 814 may beused to determine a direction of gaze. Further, a location at which gazelines projected from the user's eyes intersect the external display maybe used to determine an object at which the user is gazing (e.g. adisplayed virtual object and/or real background object). Gaze detectionsubsystem 810 may have any suitable number and arrangement of lightsources and image sensors. In other examples, gaze detection subsystem810 may use any other suitable gaze tracking technology, or may beomitted.

Display system 800 also may include additional sensors, as mentionedabove. For example, display system 800 may include non-imaging sensor(s)816, examples of which may include but are not limited to anaccelerometer, a gyroscopic sensor, a global positioning system (GPS)sensor, and an inertial measurement unit (IMU). Such sensor(s) may helpto determine the position, location, and/or orientation of the displaydevice within the environment, which may help provide accurate 3Dmapping of the real-world environment for use in displaying image dataappropriately in an augmented reality setting.

Motion sensors, as well as microphone(s) 808 and gaze detectionsubsystem 810, also may be employed as user input devices, such that auser may interact with the display system 800 via gestures of the eye,neck and/or head, as well as via verbal commands. It will be understoodthat sensors illustrated in FIG. 8 are shown for the purpose of exampleand are not intended to be limiting in any manner, as any other suitablesensors and/or combination of sensors may be utilized.

Display system 800 further includes one or more speaker(s) 818, forexample to provide audio outputs to a user for user interactions.Display system 800 further includes a controller 820 having a logicsubsystem 822 and a storage subsystem 824 in communication with thesensors, gaze detection subsystem 810, display subsystem 804, and/orother components. Storage subsystem 824 comprises instructions storedthereon that are executable by logic subsystem 822, for example, toreceive and interpret inputs from the sensors, to identify location andmovements of a user, to identify real objects in an augmented realityfield of view and present augmented reality imagery therefore, to detectobjects located outside a field of view of the user, and to presentindications of positional information associated with objects locatedoutside the field of view of the user, among other tasks.

Logic subsystem 822 includes one or more physical devices configured toexecute instructions. For example, the logic subsystem may be configuredto execute instructions that are part of one or more applications,services, programs, routines, libraries, objects, components, datastructures, or other logical constructs. Such instructions may beimplemented to perform a task, implement a data type, transform thestate of one or more components, achieve a technical effect, orotherwise arrive at a desired result.

The logic subsystem may include one or more processors configured toexecute software instructions. Additionally or alternatively, the logicsubsystem may include one or more hardware or firmware logic subsystemsconfigured to execute hardware or firmware instructions. Processors ofthe logic subsystem may be single-core or multi-core, and theinstructions executed thereon may be configured for sequential,parallel, and/or distributed processing. Individual components of thelogic subsystem optionally may be distributed among two or more separatedevices, which may be remotely located and/or configured for coordinatedprocessing. Aspects of the logic subsystem may be virtualized andexecuted by remotely accessible, networked computing devices configuredin a cloud-computing configuration.

The storage subsystem 824 includes one or more physical devicesconfigured to hold instructions executable by the logic subsystem toimplement the methods and processes described herein. When such methodsand processes are implemented, the state of the storage subsystem 824may be transformed—e.g., to hold different data.

The storage subsystem 824 may include removable and/or built-in devices.The storage subsystem 824 may include optical memory (e.g., CD, DVD,HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM,EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive,floppy-disk drive, tape drive, MRAM, etc.), among others. Storagesubsystem 824 may include volatile, nonvolatile, dynamic, static,read/write, read-only, random-access, sequential-access,location-addressable, file-addressable, and/or content-addressabledevices.

It will be appreciated that the storage subsystem 824 includes one ormore physical devices. However, aspects of the instructions describedherein alternatively may be propagated by a communication medium (e.g.,an electromagnetic signal, an optical signal, etc.) that is not held bya physical device for a finite duration.

Aspects of the logic subsystem 822 and the storage subsystem 824 may beintegrated together into one or more hardware-logic components. Suchhardware-logic components may include field-programmable gate arrays(FPGAs), program- and application-specific integrated circuits(PASIC/ASICs), program- and application-specific standard products(PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logicdevices (CPLDs), for example.

The see-through display subsystem 804 may be used to present a visualrepresentation of data held by storage subsystem 824. This visualrepresentation may take the form of a graphical user interface (GUI)comprising volumetric image data. As the herein described methods andprocesses change the data held by the storage subsystem, and thustransform the state of the storage subsystem, the state of see-throughdisplay subsystem 804 may likewise be transformed to visually representchanges in the underlying data. The see-through display subsystem 804may include one or more display devices utilizing virtually any type oftechnology. Such display devices may be combined with the logicsubsystem 822 and/or the storage subsystem 824 in a shared enclosure, orsuch display devices may be peripheral display devices.

The communication subsystem 826 may be configured to communicativelycouple the display system 800 with one or more other computing devices.The communication subsystem 826 may include wired and/or wirelesscommunication devices compatible with one or more differentcommunication protocols. As non-limiting examples, the communicationsubsystem 826 may be configured for communication via a wirelesstelephone network, or a wired or wireless local- or wide-area network.In some embodiments, the communication subsystem 826 may allow displaysystem 800 to send and/or receive messages to and/or from other devicesvia a network such as the Internet.

It will be appreciated that the depicted display system 800 is describedfor the purpose of example, and thus is not meant to be limiting. It isto be further understood that the display system may include additionaland/or alternative sensors, cameras, microphones, input devices, outputdevices, etc. than those shown without departing from the scope of thisdisclosure. For example, the display system 800 may be implemented as avirtual realty display system rather than an augmented reality system.Additionally, the physical configuration of a display device and itsvarious sensors and subcomponents may take a variety of different formswithout departing from the scope of this disclosure. Further, it will beunderstood that the methods and processes described herein may beimplemented as a computer-application program or service, anapplication-programming interface (API), a library, and/or othercomputer program product. Such computer program products may beexecutable locally on the display system 800 or other suitable displaysystem, or may be executable remotely on a computing system incommunication with the display system 800.

Another example provides, on an augmented reality computing devicecomprising a display, a method, comprising obtaining depth datarepresenting a real-world scene, identifying a real-world surface of thereal-world scene via the depth data, obtaining volumetric image data andsurface image data, the volumetric image data configured to appear asbeing located in a volume behind the real-world surface, and the surfaceimage data corresponding spatially to the real-world surface, receivinga user input configured to remove an area of the surface image data, anddisplaying at least a portion of the volumetric image data in a regionin which the area of surface image data was removed. The method mayadditionally or alternatively include obtaining the volumetric imagedata and the surface image data by constructing the volumetric imagedata and the surface image data based on the depth data representing thereal-world scene. The method may additionally or alternatively includerendering the volumetric image data prior to receiving the user input.The method may additionally or alternatively include rendering thevolumetric image data by rendering the volumetric image data based onreal-world lighting effects. The method may additionally oralternatively include rendering the volumetric image data by renderingthe volumetric image data based on virtual lighting effects. The methodmay additionally or alternatively include applying a shadow cast by thereal-world surface onto the volumetric image data. The method mayadditionally or alternatively include applying virtual lighting effectswithin the shadow cast by the real-world surface. The method mayadditionally or alternatively include, after receiving the user inputconfigured to remove the area of surface image data, applying real-worldlighting effects illuminating the shadow on displayed volumetric imagedata. The method may additionally or alternatively include rendering thevolumetric image data after receiving the user input. The method mayadditionally or alternatively include obtaining the volumetric imagedata by obtaining the volumetric image data from a remote device. Themethod may additionally or alternatively include applying virtuallighting effects to the surface image data. In the example, thevolumetric image data may additionally or alternatively include blocks.The method may additionally or alternatively include acquiring imagedata of the real-world scene, and wherein obtaining the depth datarepresenting the real-world scene comprises obtaining the depth databased on the image data.

Another example provides an augmented reality display system, comprisinga display, a logic subsystem, and a storage subsystem comprisinginstructions executable by the logic subsystem to obtain athree-dimensional mesh representation of a real-world scene, identify areal-world surface of the real-world scene via the three-dimensionalmesh representation, obtain volumetric image data and surface imagedata, the volumetric image data configured to appear as being located ina volume behind the real-world surface, and the surface image datacorresponding spatially to the real-world surface, receive a user inputconfigured to remove an area of the surface image data, and display atleast a portion of the volumetric image data in a region in which thearea of the surface image data was removed. The instructions executableto obtain the volumetric image data and the surface image data mayadditionally or alternatively include instructions executable toconstruct the volumetric image data and the surface image data based onthe three-dimensional mesh representation of the real-world scene. Theinstructions may additionally or alternatively be executable to renderthe volumetric image data at the computing device prior to receiving theuser input. The instructions may additionally or alternatively beexecutable to render the volumetric image data after receiving the userinput. The instructions executable to obtain the volumetric image datamay additionally or alternatively include instructions executable toobtain the volumetric image data from a remote device.

Another example provides a head-mounted display device, comprising asee-through display, an image sensor system, a logic subsystem, and astorage subsystem comprising instructions executable by the logicsubsystem to acquire image data of a real-world scene via the imagesensor system, obtain a three-dimensional mesh representation of thereal-world scene based on the image data, identify a real-world surfaceof the real-world scene via the three-dimensional mesh representation,obtain rendered volumetric image data and surface image data, therendered volumetric image data configured to appear as being located ina volume behind the real-world surface, and the surface image datacorresponding spatially to the real-world surface, receive a user inputconfigured to remove an area of the surface image data, and display atleast a portion of the rendered volumetric image data in a region inwhich the area of the surface image data was removed. The instructionsexecutable to obtain the rendered volumetric image data may additionallyor alternatively include instructions executable to render volumetricimage data based on real-world lighting effects.

It will be understood that the configurations and/or approachesdescribed herein are presented for example, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnonobvious combinations and subcombinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

The invention claimed is:
 1. On an augmented reality computing devicecomprising a display, a method, comprising: obtaining depth datarepresenting a real-world scene; identifying a plurality of real-worldsurfaces of the real-world scene via the depth data; constructingvolumetric image data and surface image data based on the depth data,the volumetric image data comprising a plurality of volumetric imagedata units filling at least a portion of a volume behind a real-worldsurface of the plurality of real-world surfaces, and the surface imagedata corresponding spatially to the real-world surface; receiving afirst user input configured to remove an area of the surface image data;displaying at least a first portion of the volumetric image data in afirst region in which the area of the surface image data was removed;receiving a second user input configured to remove the first portion ofthe volumetric image data by removing one or more of the volumetricimage data units of the volumetric image data; and displaying at least asecond portion of the volumetric image data in a second region revealedby removal of the one or more volumetric image data units.
 2. The methodof claim 1, further comprising rendering the volumetric image data priorto receiving the first user input.
 3. The method of claim 2, whereinrendering the volumetric image data comprises rendering the volumetricimage data based on real-world lighting effects.
 4. The method of claim2, wherein rendering the volumetric image data comprises rendering thevolumetric image data based on virtual lighting effects.
 5. The methodof claim 2, further comprising applying a shadow cast by a real-worldobject onto the volumetric image data.
 6. The method of claim 5, furthercomprising applying virtual lighting effects within the shadow.
 7. Themethod of claim 5, further comprising, after receiving the first userinput configured to remove the area of the surface image data, applyingreal-world lighting effects illuminating the shadow on displayedvolumetric image data.
 8. The method of claim 1, further comprising oneor more of rendering the first portion of the volumetric image dataafter receiving the first user input and rendering the second portion ofthe volumetric image data after receiving the second user input.
 9. Themethod of claim 1, wherein constructing the volumetric image datacomprises obtaining the volumetric image data from a remote device. 10.The method of claim 1, further comprising applying virtual lightingeffects to the surface image data.
 11. The method of claim 1, whereinthe volumetric image data comprises blocks.
 12. The method of claim 1,further comprising acquiring image data of the real-world scene, andwherein obtaining the depth data representing the real-world scenecomprises obtaining the depth data based on the image data.
 13. Anaugmented reality display system, comprising: a display; a logicsubsystem; and a storage subsystem comprising instructions executable bythe logic subsystem to obtain a three-dimensional mesh representation ofa real-world scene, identify a plurality of real-world surfaces of thereal-world scene via the three-dimensional mesh representation,construct volumetric image data and surface image data, the volumetricimage data comprising a plurality of volumetric image data units fillingat least a portion of a volume behind a real-world surface of theplurality of real-world surfaces, and the surface image datacorresponding spatially to the real-world surface, receive a first userinput configured to remove an area of the surface image data, display atleast a first portion of the volumetric image data in a first region inwhich the area of the surface image data was removed, receive a seconduser input configured to remove the first portion of the volumetricimage data by removing one or more of the volumetric image data units ofthe volumetric image data, and display at least a second portion of thevolumetric image data in a second region revealed by removal of the oneor more volumetric image data units.
 14. The augmented reality displaysystem of claim 13, wherein the instructions executable to construct thevolumetric image data and the surface image data comprise instructionsexecutable to construct the volumetric image data and the surface imagedata based on the three-dimensional mesh representation of thereal-world scene.
 15. The augmented reality display system of claim 13,wherein the instructions are further executable to render the volumetricimage data prior to receiving the first user input.
 16. The augmentedreality display system of claim 14, wherein the instructions are furtherexecutable to render the volumetric image data after receiving the firstuser input.
 17. The augmented reality display system of claim 13,wherein the instructions executable to construct the volumetric imagedata comprise instructions executable to obtain the volumetric imagedata from a remote device.
 18. A head-mounted display device,comprising: a see-through display; an image sensor system; a logicsubsystem; and a storage subsystem comprising instructions executable bythe logic subsystem to acquire image data of a real-world scene via theimage sensor system, obtain a three-dimensional mesh representation ofthe real-world scene based on the image data, identify a plurality ofreal-world surfaces of the real-world scene via the three-dimensionalmesh representation, construct rendered volumetric image data andsurface image data, the rendered volumetric image data comprising aplurality of volumetric image data units filling at least a portion of avolume behind a real-world surface of the plurality of real-worldsurfaces, and the surface image data corresponding spatially to thereal-world surface, receive a first user input configured to remove anarea of the surface image data, display at least a first portion of therendered volumetric image data in a first region in which the area ofthe surface image data was removed, receive a second user inputconfigured to remove the first portion of the rendered volumetric imagedata by removing one or more of the volumetric image data units of therendered volumetric image data, and display at least a second portion ofthe rendered volumetric image data in a second region revealed byremoval of the one or more volumetric image data units.
 19. Thehead-mounted display device of claim 18, wherein the instructionsexecutable to construct the rendered volumetric image data compriseinstructions executable to render volumetric image data based onreal-world lighting effects.